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EXECUTIVE  SUMMARY 


Over  the  years  many  meteorologists  have  declared  that  improvement  in  weather  fore¬ 
casting  has  been  negligible  in  spite  of  significant  improvements  in  meteorological 
observations,  data  communications,  and  analysis  and  prognosis  procedures.  In  the 
most  recent  10  to  15  years,  concern  increased  as  it  became  clear  that  the  accuracy  of 
centrally  produced  prognoses  steadily  improved,  while  accuracy  of  weather  forecasting, 
as  determined  by  several  studies,  showed  negligible  improvement.  At  the  same  time, 
other  studies  showed  improvement.  Since  improved  weather  forecasting  is  a  stated  goal 
of  numerous  projects  involving  relatively  large  monetary  outlays,  a  study  was  begun 
to  examine  available  data  to  independently  determine  whether  weather  forecasting  was 
improving.  We  found  that  it  was. 

More  than  100  articles  applicable  to  the  subject  were  reviewed.  Of  these  only  a  few 
contained  data.  These  studies  were  individually  summarized  and  assessed  in  the 
second  section.  The  overall  finding  was  that  there  was  significant  improvement  in 
forecasting  skill  for  nearly  every  weather  element.  In  nearly  every  case,  the 
trends  were  statistically  significant.  It  is  clear  that  weather  forecasting  overall 
has  improved  in  recent  years  at  a  rate  that  is  generally  in  agreement  with  the 
improvements  obtained  by  the  numerical  weather  prediction  models.  Some  studies 
definitely  indicated  no  improvement  for  weather  elements  for  which  other  studies 
showed  significant  improvement.  Accordingly,  it  became  important  to  consider  other 
factors  that  might  be  important  in  determining  forecasting  skill.  This  was  done  in 
Section  III. 

Several  factors  commonly  cited  as  being  significant  in  determining  forecasting  skill 
are  meteorological  education,  forecasting  experience,  forecaster  interest  and  aptitude, 
and  weather  station  operations  (systematic  procedures  and  forecaster  workload). 
Generally  few  studies  are  available  that  address  these  subjects.  Those  available 
indicate  that  beyond  an  initial  2  to  3  months  meteorological  education  and  1  to  2 
months  experience,  there  is  negligible  improvement  in  forecasting  skill.  Some 
authors  asserted  that  forecasters  having  greater  education  and  experience  forecast 
the  rarer  events  more  accurately;  however,  this  result  could  not  be  unequivocally 
established.  Furthermore,  much  of  this  experience  could  be  learned  through  study  of 
forecaster  references.  Forecaster  interest  and  aptitude  were  accepted  as  important. 

In  general,  however,  forecasters  are  not  selected  through  achievement  of  high  ratings 
in  these  areas.  We  then  considered  some  aspects  of  weather  station  operations. 

Overall,  systematization  of  weather  station  forecasting  has  not  been  successful. 
Problems  cited  by  Willett  (1951)  remain  extant.  Forecaster  workload  was 
discussed;  it  was  noted  that  the  time  generally  available  for  forecasting  had  not 
increased  even  though  numerous  time  consuming  analysis  and  prognosis  procedures 
were  now  being  provided  to  the  weather  stations  by  centrals. 

Forecaster  motivation  was  identified  as  a  factor  which  appears  to  be  a  component  of 
forecasting  skill.  This  conclusion  was  based  on  the  assumption  that  the  rapid  growth 
of  commercial  weather  services  was  an  indication  that  financial  motivation  was 
effective,  and  on  personal  experience  in  working  with  Air  Weather  Service  forecasters 
for  a  number  of  years.  Air  Weather  Service  has  been  emphasizing  forecasting  per¬ 
formance,  and  the  performance  results  have  shown  large  increases  in  skill.  Military 
experience  has  been  that  the  single  most  important  factor  in  successful  weather 
station  operations  was  the  person  in  charge.  Experiments  that  could  be  conducted 
might  include  subjects  such  as  measuring  the  performance  of  highly  experienced, 
highly  educated,  minimally  experienced,  and  minimally  educated  forecasters;  and 
the  effect  on  forecasting  skill  of  less  workload  and  bonus  pay  for  performance. 

In  general,  the  overall  result  of  this  effort  was  a  revalidatior.  of  the  cost 
effectiveness  of  Air  Weather  Service  emphasis  on  a  forecaster  force  mostly 
composed  of  enlisted  personnel.  It  indicates  that  corporately  we  may  be  overly 
concerned  with  forecasting  experience  and  formal  training,  and  insufficiently 
concerned  with  forecasting  motivation,  forecaster  selection,  and  enhancing  the 
thought  processes  required  for  forecasting.  The  AMS  emphasis  on  forecasting 
skill  apparently  was  effective,  and  should  bo  continued.  Numerical  weather 
prediction  efforts  should  continue,  and  are  likely  to  improve  weather  forecasting 
over  the  next  few  years,  further  improvements  likely  will  be  achieved  through 
continued  emphasis  on  weather  forecasting  skill  and  on  effective  programs  to 
encourage  forecasters  to  routinely  do  the  hard  mental  work  with  basic  meteorology 
that  is  essential  for  good  weather  forecasting. 
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Throughout  this  century,  meteorologists  have  been  striving  to  improve  the  accuracy 
of  weather  forecasting.  The  consensus  has  been  that  significant  improvement  could 
be  obtained  by  increasing  the  density  and  frequency  of  weather  observations,  improv¬ 
ing  the  flow  of  data  for  more  rapid  analysis,  and  developing  more  powerful  forecast¬ 
ing  techniques  (Byers,  1944;  Willett,  1951;  Kreitzberg,  1976;  Pielke,  1977;  AMS, 
1979;  and  many  others).  Large  resources  have  been  expended  toward  the  attainment  of 
these  goals.  Although  remarkable  advances  have  been  rade  in  all  these  areas  and  in 
synoptic  meteorology,  many  feel  the  improvements  in  forecasting  the  weather  (clouds, 
winds,  precipitation,  and  obstructions  to  vision)  have  been  small  (Holzman,  1947; 
Dunn,  1951;  Willett,  1951;  Sanders,  1973,  1979,  Bosart,  1975;  Jensen,  1975;  Ramage, 
1976,  1978;  Cook  and  Smith,  1977;  Fawcett,  1977;  Brown,  1978;  Shuman,  1978;  Smith, 
1979;  and  Rinne  and  Frisk,  1979) .  A  study  was  undertaken  to  evaluate  the  trends  in 
forecasting  performance,  and  to  assess  factors  affecting  forecasting  skill.  The 
approach  was  to  review  forecasting  skill  information  published  from  the  late  1940s 
to  the  present.  Forecasting  skill  achieved  through  both  objective  (numerical) 
techniques  and  forecasters'  subjective  improvements  were  considered.  A  summary  of 
the  1950  weather  forecasting  state-of-the-art  is  given  as  background. 

Weather  Forecasting  State-of-the-Art  in  1950 

Weather  forecasting  generally  is  considered  a  three  step  process:  analyzing,  prog¬ 
nosticating,  and  determining  the  forecast  weather.  In  1950,  analyses  were  considered 
generally  adequate,  although  additional  observing  sites  were  needed,  primarily  in 
remote  and  ocean  areas.  Prognoses  for  periods  up  to  36  hours  were  considered 
essentially  satisfactory.  Determining  the  forecast  weather  from  these  prognoses  was 
dona  by  subjectively  considering  the  effects  of  various  dynamical  and  physical 
processes  which  might  have  acted  or  be  acting  on  the  air  mass.  Willett  (1951) 
stated  that  negligible  progress  had  been  made  in  the  previous  forty  years  in  deter¬ 
mining  the  forecast  weather  in  spite  of  greatly  improved  instruments  and  increased 
numbers  of  weather  observing  locations.  Reasons  he  gave  included  a  large  increase 
in  numbers  of  weather  forecasters  without  the  necessary  aptitude,  training,  and 
experience;  lack  of  standard  forecasting  procedures;  and  lack  of  authoritative 
direction.  Willett  suggested  that  improvement  in  determining  the  forecast  weather 
was  possible  by  improving  the  reliability  and  distribution  of  weather  observations, 
standardizing  techniques,  and  improving  the  process  of  selecting  and  training 
weather  forecasters. 

Fvcichelderfer  (1951)  noted  that  despite  remarkable  advances  in  understanding  mete¬ 
orology,  comparable  improvements  in  accuracy  of  weather  forecasts  had  not  occurred. 
He  suggested  that  forecasters  in  general  had  not  been  able  to  assimilate  the  recent 
large  increases  in  knowledge,  noting  that  ever  more  demanding  forecast  schedules 
left  little  time  or  incentive  to  keep  up  with  voluminous  new  meteorological  litera¬ 
ture.  Others  also  had  noted  substantial  improvements  made  in  understanding  meteor¬ 
ology,  and  in  improving  instruments,  observation  frequency,  serological  observation 
quality  and  frequency,  numbers  of  reporting  stations,  advances  in  prognostic  tech¬ 
niques,  and  the  data  collection  system.  These  improvements  were  often  perceived  as 
not  accompanied  by  comparable  increases  in  accuracy  of  determining  the  forecast 
weather. 

Forecasting  weather  was  generally  perceived  as  requiring  highly  educated,  trained, 
and  experienced  personnel  with  special  aptitudes.  Research  was  directed  toward 
improved  understanding  of  atmospheric  processes,  with  little  study  being  done  of  how 
to  improve  forecasting  skill,  assuming  increased  skill  would  accompany  increased 
knowledge.  At  this  time,  economics  and  science  both  led  toward  increased  central¬ 
ization  of  weather  forecasting  activities  resulting  in  major  changes  in  station 
forecasting  procedures.  This  and  the  impact  of  computers  and  satellites  will  be 
discussed  next. 

Centralization 


In  the  early  1940s,  rapid  expansion  of  requireswnts  for  meteorological  services 
forced  consideration  of  new  ways  to  meet  needed  service.  It  was  soon  recognised 
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that  it  would  be  prohibitively  costly  to  staff  all  weather  stations  with  enough 
people  to  plot  and  analyze  all  charts  needed  for  quality  forecasting.  Consequently, 
analysis  centers  were  established  and  manned  by  the  best  qualified  scientists.  The 
analyses  were  disseminated  by  facsimile,  making  numerous  high  quality  analyses 
available  for  production  of  needed  forecasts  (Holzmann,  1947) .  The  advantages  of 
analysis  centralization  soon  led  to  prognosis  centralization. 

There  were  several  reasons  for  centralization.  During  World  War  ZI,  the  number  of 
stations  reporting  increased  greatly,  observations  became  routinely  available  to 
greater  altitudes,  and  a  great  deal  was  learned  about  weather  map  analysis  and 
prognosis.  The  commonly  accepted  procedures  for  analysis  and  prognosis  had  expanded 
so  much  that  they  could  not  be  accomplished  in  the  weather  stations  with  available 
manning  and  meteorological  expertise.  At  the  same  time  requirements  for  weather 
services,  such  as  more  frequent  observations  (meteorological  watch)  and  increased 
tailoring  of  weather  products  and  services  were  increasing. 

The  procedures  used  for  preparing  prognostic  maps  out  to  36  hours  essentially  were 
based  on  extrapolation,  subjectively  modified  based  on  the  forecaster's  experience 
and  understanding  of  applicable  theory  (Godske,  et  al.,  1957) .  In  general  the 
entire  process  was  highly  subjective,  including  analysis.  Sometimes  different 
central  analysis  and  forecasting  teams  produced  radically  differing  analyses  and 
prognoses  using  the  same  data  (Thompson,  1961).  However,  the  overall  accuracy  was 
considered  satisfactory.  Prognosis  verification  exhibited  a  relatively  level  trend 
from  year  to  year.  Fronts,  precipitation  areas,  and  so  forth  were  analyzed  to  an 
accu  acy  of  about  50  kilometers;  prognoses  for  24  to  36  hours  were  accurate  within 
100  to  200  kilometers  (Bundgaard,  1951).  Centralization  continued,  and  in  1955  the 
National  Weather  Analysis  Central  was  formed  at  Suitland  through  merger  of  the 
Weather  Bureau,  Air  Force,  and  Navy  Centrals  in  the  Washington,  D.  C.  area.  This 
change  in  the  weather  forecast  production  system  has  not  been  associated  with  any 
identifiable  improvement  in  forecasting  accuracy.  (See  Fawcett,  1962  and  Shuman, 
1978). 

Numerical  Weather  Prediction 


As  early  as  1950,  research  in  numerical  weather  prediction  was  active.  Many  of  the 
basic  problems  had  been  solved,  and  bigger  and  faster  computers  were  being  developed. 
By  1955,  many  had  become  convinced  that,  "The  development  of  numerical  methods  of 
weather  analysis  and  forecasting  is  one  of  the  most  significant  and  spectacular 
advances  in  the  practice  of  weather  forecasting  and  the  science  of  meteorology 
. ..."  (Thompson,  1961).  In  1955,  the  Joint  Numerical  Weather  Prediction  Unit 
began  daily  production  of  computer  generated  forecasts  for  North  America  (Fuller, 
1977).  Figure  1  shows  the  verification  data  for  both  manual  (m)  and  computer 
prepared  30/36  hour  surface  and  the  36  hour  500  millibar  prognoses.  The  dramatic 
improvement  in  score  during  1958  has  been  attrik  •  ed  to  the  barotropic  numerical 
forecasts  becoming  available  in  time  to  be  used  as  the  basis  for  the  NMC  official 
(manual)  500  millibar  prognoses.  The  surface  prognoses  show  a  modest  (12*)  improve¬ 
ment  after  1955,  and  a  small  (34)  improvement  beginning  after  1965,  reflecting 
more  complete  and  systematic  use  of  numerical  products.  (Fawcett,  1962).  The 
manual  production  of  the  500  millibar  prognosis  was  stopped  when  it  became  apparent 
that  the  computer  produced  forecast  was  about  as  good  as  the  manual  product. 

Figure  1  shows  a  substantial  improvement  in  the  performance  of  the  guidance  products. 
It  is  generally  agreed  that  in  contrast  to  the  computer,  forecasters  cannot  produce 
comparable  prognostic  charts  for  large  (hemispheric)  areas;  however,  they  can 
improve  the  prognoses  for  limited  areas  through  use  of  later  data  (Snellman,  1977; 
MacDonald,  1977;  Bosart,  1978;  Brown,  1978;  Cahir,  1978;  and  Cook  and  Smith,  1978). 

The  performance  of  the  Limited-Area  Fine  Mesh  (LFM)  prognoses  over  the  CONUS  shows 
further  improveisent.  The  practical  range  of  SI*  scores  for  the  500  millibar  charts 
is  from  about  20  (nearly  perfect)  to  about  70  (which  corresponds  to  climatology) 
(Shuman,  1978).  Thus,  in  the  twenty  odd  years  since  numerical  weather  prediction 
models  have  been  operational,  over  70  percent  of  the  possible  improvement  has  been 
achieved  for  36  bout  CONUS  500  millibar  forecasts.  This  achievement  cannot  be 
directly  Interpreted  as  an  improvement  In  weather  forecasting.  Even  with  the 
superior  accuracy  of  the  LFM,  forecasters  can  improve  the  prognoses  over  limited 


The  SI  score  (Teweles  and  Wobus,  1954)  is  a  measure  of  forecast  pressure/  height 
change  error  relative  to  the  larger  of  the  forecast  or  observed  pressure/  height 
gradient. 
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areas  as  mentioned  above.  The  surface  charts  are  not  as  accurate  as  the  500  mb 
prognoses,  and  the  moisture  and  vertical  velocity  prognoses  need  considerable 
improvement  even  though  substantial  gains  have  been  made  since  the  first  forecasts. 
The  moisture  and  vertical  motion  fields  are  now  routinely  modified  by  National 
Meteorological  Center  precipitation  and  cloud  forecasters  (Brown  and  Olson,  1978) . 

Meteorological  Satellites 

The  advent  of  the  meteorological  satellite  was  hailed  as  a  major  scientific  break¬ 
through  that  would  lead  to  significant  improvements  in  forecasting.  While  this  may 
have  occurred  in  some  instances,  we  have  found  no  published  supporting  data.  There 
are,  however,  many  new  techniques  for  obtaining  better  understanding  of  various 
meteorological  processes.  As  examples,  see  Weldon  (1979)  for  relationships  between 
cloud  patterns  and  upper  air  wind  flow,  Miller  and  McGinly  (1978)  for  relationships 
with  severe  convective  weather,  and  Gurka  (1978)  for  fog  forecasting.  At  the 
present  time,  meteorological  satellite  imagery  is  not  automatically  included  in 
most  weather  prediction  models;  much  additional  work  is  needed.  In  some  respects 
the  availability  of  satellite  data  may  have  contributed  to  the  loss  of  a  number  of 
observing  stations  for  both  surface  and  aloft,  and  created  changes  in  personnel  and 
operational  concepts  that  tended  to  counteract  some  of  the  innovative  advantages  of 
the  satellite. 

Initially  weather  satellite  information  consisted  of  visual  and  infrared  images 
with  moderate  resolution.  These  pictures  helped  meteorologists  recognize  cloud 
patterns  better,  but  the  knowledge  was  gained  incrementally  over  a  period  of  several 
years.  Thus,  no  specific  impact  was  noted  in  the  published  verification  data.  In 
time,  polar  orbiting  satellites  produced  better  images,  permitting  observation  of 
smaller  details.  Images  became  available  at  12  hour  intervals,  but  these  times  did 
not  correspond  to  times  when  other  data  were  available.  As  a  result,  the  forecasters 
had  to  do  a  lot  of  manual  correlation  (eyeballing),  and  this  translation  often  was 
not  accurate.  Nevertheless,  the  satellites  improved  the  forecasters'  understanding 
of  cloud  distributions  and  analyses  over  data  sparse  regions. 

Geostationary  satellites  provided  an  immense  increase  of  data.  Every  half  hour 
visual  and/or  infrared  images  were  produced  for  about  60°  latitude  from  the  sub¬ 
satellite  point  on  the  Earth's  equator.  Temporal  correlation  with  synoptic 
observations  was  now  possible,  permitting  the  inclusion  of  satellite  information  on 
analysts  work  charts.  In  spite  of  this,  the  data  continued  to  be  produced  at 
different  scales  and  manually  transferred  to  analyses.  The  frequent  images  over 
the  same  geographic  area  added  a  new  capability.  Rapidly  changing  cloud  patterns 
are  associated  with  important  and  severe  weather.  Most  of  these  events  are  now 
identified  witnin  the  hour  so  that  forecasters  in  facilities  with  satellite  data 
are  much  more  aware  of  developing  weather  changes  than  they  were  in  the  presatellite 
era.  One  indication  of  the  operational  usefulness  of  the  GOES  data  is  provided  by 
a  short  unpublished  study  done  by  the  7th  Weather  Wing  Aerospace  Sciences  personnel. 
The  performance,  as  measured  by  the  Air  Weather  Service  skill  score  (possible 
percentage  improvement  over  persistence)  at  three  stations  was  compared  before  and 
after  getting  routine  access  to  GOES  data.  The  improvement  averaged  15  percent  at 
3  hours  and  54  percent  at  6  hours.  These  improvements  occurred  at  a  time  when 
wing-wide  forecaster  experience  and  manning  were  low,  and  the  skill  score  at  other 
wing  units  increased  less  than  3  percent  during  the  same  period.  Otherwise,  no 
references  were  found  which  document  improvements  in  performance  attributad  to 
satellite  images.  Perhaps  there  has  been  no  discernable  improvement  overall. 

Techniques  have  been  developed  to  make  temperature  soundings  of  the  atmosphere  from 
the  polar  orbiting  satellites.  The  first  versions  were  not  successful,  but  a 
second  generation  is  now  in  operation.  The  soundings  are  not  as  structured  as  the 
radiosonde,  however,  the  main  features  are  sufficiently  accurate.  The  satellite 
soundings  are  helpful  in  sparse  data  regions,  and  recent  tests  (Ohring,  1979) 
indicate  an  improvement  of  1  to  2  points  in  the  standard  scoring  system  (SI)  for 
prognoses  in  the  48  hour  range.  At  present  rates  of  improvement,  this  is  equivalent 
to  a  2  to  5  year  advance  of  the  state  of  the  art  or  a  2b  to  5  hour  extension  of 
forecast  capability  in  the  48  hour  forecast  range. 
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Surmary 

During  the  1940s  significant  increases  were  made  in  the  numbers  of  surface  and 
upper  air  observations,  and  in  the  speed  with  which  these  date,  were  transmitted. 

The  additional  data  led  to  rapid  improvements  in  weather  analysis  and  to  significant 
improvements  in  forecasting  techniques.  The  additional  data  /^nd  new  techniques 
caused  weather  station  manpower  requirements  to  increase  substantially;  this 
problem  was  solved  through  centralization  of  weather  map  analysis  and  prognosis. 
Personnel  in  the  weather  centrals  were  selected  for  their  meteorological  knowledge 
and  skill  in  weather  map  analysis  and  prognosis,  and  the  process  of  expertise 
decay  in  base  weather  station  began.  It  is  assumed  that  the  forecasting  performance 
improved  during  the  1940s,  but  verification  data  were  not  found.  By  the  early 
1950s,  forecasting  performance,  as  measured  by  verification  of  the  30  to  36  hour 
surface  and  500  millibar  prognoses,  was  essentially  level.  Subsequently,  significant 
improvements  were  achieved,  primarily  through  advances  in  numerical  weather  prediction. 
Analysis  and  prognosis  of  clouds  improved  with  increased  availability  and  use  of 
weather  satellites;  however,  the  use  of  weather  satellite  data  remains  a  developing 
technology.  Although  many  specific  techniques  for  analyzing  weather  satellite 
imagery  have  been  idenvified,  much  remains  to  be  done  before  full  use  of  such 
systems  can  be  achieved.  Overall,  then,  substantial  improvements  were  made  during 
the  1940s  and  1950s  in  tnose  areas  recommended  by  meteorologists  as  being  likely  to 
improve  forecasting  —  density  and  frequency  of  weather  observations,  faster  data 
flow  and  analysis,  and  morn  powerful  forecasting  techniques.  As  a  result,  sub¬ 
stantial  improvements  in  weather  map* prognoses  were  achieved.  The  next  section 
considers  the  corresponding  changes  in  forecasting  the  weather. 


FORECASTING  PERFORMANCE 

Forecasting  performance,  as  measured  by  verification  of  30  to  36  hour  surface  and 
500  millibar  prognoses,  became  essentially  level  in  the  early  1950s.  Subsequently , 
significant  improvements  were  achieved  through  advances  in  numerical  weather 
prediction.  In  spite  of  large  improvements  in  the  accuracies  of  the  prognostic 
maps,  some  suggest  that  little  improvement  has  been  achieved  in  forecasting  clouds 
and  weather.  Several  studies  will  be  discussed. 

The  Massachusetts  Institute  of  Technology  (MIT)  Forecasting  Game 

The  MIT  departmental  weather  forecasting  program  was  developed  to  provide  instruction 
in  the  ways  of  the  real  atmosphere  (Sanders,  1973).  Forecasts  were  made  weekdays 
for  the  probability  that  the  minimum  temperature  would  exceed  the  normal  minimum 
for  the  date  and  probability  of  at  least  .01  inches  of  precipitation  at  Logan 
International  Airport,  Boston,  MA.  Probability  forecasts  of  distributions  of 
temperature  and  precipitation  amounts  were  also  made.  Climatological  mean  values 
were  used  for  guidance  and  as  control  forecasts.  The  forecasts  were  for  four 
consecutive  24  hour  periods,  beginning  at  1800  GMT  of  the  current  day.  The  first 
period  was  called  the  24  hour  forecast  and  the  last  the  96  hour  forecast.  Fore- 
casts  had  to  be  completed  not  later  than  1330  local  time  (during  winter  this 
completion  time  was  30  minutes  after  start  of  the  first  forecast  period).  Most 
forecast  deliberation  occurred  between  1200  and  1330.  Facsimile  and  teletype  data 
were  available  for  study  at  all  hours.  The  program  was  open  to  anyone,  but  the 
participants  were  mainly  graduate  meteorology  students  and  undergraduates  taking 
meteorology  courses.  Each  day  a  consensus  forecast  was  made;  it  was  the  average 
of  those  participating  for  the  given  day,  forecast  element,  and  time  period.  Each 
forecaster  was  scored  against  the  consensus  forecasts,  and  only  for  the  forecasts  he 
made.  The  scores  for  the  minimum  temperature  forecasts  were  absolute  error;  the 
Brier  score  for  precipitation  and,  the  Murphy's  Ranked  Probability  Score  for  the 
forecast  temperature  and  precipitation  amount  class  probability  distributions. 

The  trends  for  consensus  forecasts  of  minimum  temperature  and  probability  of  six 
classes  of  precipitation  amount  forecasts  are  shown  in  Figures  2  and  3  (Sanders, 
1979).  The  performance  showed  negligible  improvement  for  12  years-* a  stark  contrast 
to  the  significant  improvement  shown  in  Figure  1  for  surface  and  500  millibar 
,  .ograms;  however,  the  level  of  performance  was  hi^h,  40  to  50%  improvestent  over 
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climatology  for  the  first  period.  The  lack  of  an  improved  trend  was  interpreted  by 
Sanders  (1973)  as  resulting  from  large  errors  in  the  first  period  forecasts  due  to 
mesoscale  effects  (shallow  fronts,  land-sea  contrast,  convective  showers,  and 
inaccurate  timing  of  non-convective  precipitation).  Consensus  performance,  by 
remaining  more  or  less  steady  in  the  absolute  Bense,  actually  deteriorated  relative 
to  the  rising  state  of  the  art.  It  waB  suggested  that  this  was  due  to  inexperience, 
Sanders'  own  performance  was  essentially  constant  relative  to  consensus,  indicating 
a  decrease  of  his  skill  relative  to  the  state  of  the  art,  and  discounting  experience 
as  the  reason  for  the  relative  deterioration  of  the  consensus  forecasts.  It  is 
possible  that  the  model  skill  was  too  small  to  be  seen  in  the  excellert  forecasting 
performance  achieved,  but  model  skill  improvements  eventually  should  be  reflected 
in  the  longer  range  forecasts. 

The  performance  of  the  consensus  forecast  relative  to  objective  guidance  produced 
by  NMC  using  the  1200Z  data  base  was  also  evaluated.  The  skill  of  the  guidance 
improved  rapidly  during  the  late  1960s,  but  was  nearly  constant  after  1972.  The 
percent  improvement  of  the  consensus  forecast  over  guidance  was  about  18%  overall 
for  minimum  temperature  and  40%  for  quantitative  precipitation  forecasts.  While 
the  sample  size  used  in  these  evaluations  was  not  large  enough  to  be  conclusive,  it 
certainly  is  thought  provoking.  Sanders  (1973)  suggested  further  studies  using 
broader  data  base;  a  similar  program  was  established  at  the  State  University  of  New 
York  at  Albany  (SUNYA) . 

The  SUNYA  Forecasting  Game 

The  forecasting  game  at  the  State  University  of  New  York  at  Albany  was  essentially 
the  same  as  the  one  at  MIT  (Bosart,  1975) .  However,  only  two  forecasts  were  made 
for  each  24  hour  period  -  the  probability  that  the  minimum  temperature  will  be 
below  the  climatological  minimum,  and  the  probability  that  there  will  be  measurable 
precipitation.  The  participants  were  mainly  graduate  students.  Initially  there 
was  no  forecasting  experience  other  than  the  author's  participation  in  the  MIT  game 
for  five  years.  The  results  are  shown  in  Figures  4  and  5.  Again  the  level  of 
skill  was  high,  a  40  to  50  percent  improvement  over  climatology  for  the  first 
period.  There  is  no  trend  for  the  consensus  temperature  forecasts  and  for  the 
third  and  fourth  day  precipitation  forecasts.  A  marked  improvement  was  shown  for 
the  consensus  24  hour  precipitation  forecast,  and  small  improvement  was  shown  for 
the  consensus  48  hour  and  all  hour  forecasts.  Among  the  factors  Bosart  listed  as 
contributing  to  these  results  were:  (a)  The  experience  level  of  the  student 
forecasters  had  increased  significantly  by  the  spring  1972  semester,  (b)  The 
students  reduced  use  of  large  departures  ftom  the  climatology  forecasts  for  the  72 
and  96  hour  forecasts,  and  (c)  The  introduction  of  the  LFM  and  model  output 
statistics  (MOS) .  The  LFM  used  1200Z  observations,  but  was  projected  for  only  48 
hours.  The  MOS  data  weie  derived  from  the  previous  evening's  00Z  data  base,  and 
projected  further  in  time.  Bosart' s  personal  results  are  shown  in  Figure  6  in 
terms  of  percentage  improvement  over  consensus.  Like  Sanders',  there  is  no  clear 
trend  overall  for  either  precipitation  or  temperature.  In  general,  this  study 
supports  the  results  reported  by  Sanders  (1967,  1973)  of  no  long  term  improving 
trend. 

Forecasts  at  Louisville,  Kentucky 

Public  probability  of  precipitation  and  maximum/minimum  temperature  forecasts  for 
Louisville,  XY  were  evaluated  by  Cook  and  Smith  (1977).  The  forecasts  were  for  12 
hour  periods  over  a  ten  year  span  (1966  -  1975).  Each  period  was  evaluated 
separately  and  compared  to  guidance  forecasts  issued  by  the  National  Meteorological 
Center.  There  was  little  change  in  the  quality  of  precipitation  forecasts 
issued  even  though  central  guidance  had  improved;  however,  forecasters  were  able  to 
recognise  instances  where  the  guidance  forecasts  were  significantly  in  error  (CT 
30%) .  In  such  cases,  improved  forecasts  were  issued  for  about  70  percent  of  the 
cases  for  the  first  period,  60  percent  for  the  second  and  third  forecast  periods. 

The  overall  quality  of  the  temperature  forecasts  also  exhibited  little  change  over 
the  ten  year  period.  Temperature  changes  of  10°F  or  larger  were  forecast  slightly 
Letter  by  the  WSFO  in  the  first  and  second  periods.  Generally,  these  results 
confirm  Sanders'  (1973)  suggestion  that,  in  spite  of  improvements  in  meteorology, 
weather  forecasting  had  not  improved  through  1975. 
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National  Weather  Service  Precipitation  Forecasts 

Charba  and  Klein  (1980a  and  b)  surveyed  the  accuracy  of  different  types  of  preciDita- 
tion  forecasts  produced  in  the  National  Weather  Service.  The  study  included 
objective  and  subjective  guidance  forecasts  prepared  at  the  National  Meteorological 
Center  (NMC) ,  and  the  local  forecasts  issued  by  the  forecast  offices  using  the 
guidance  forecasts. 

Categorical  Forecasts  of  Measurable  Precipitation 

The  NMC  forecaster  who  prepares  the  precipitation  areas  appearing  on  the  national 
surface  prognostic  charts  assigns  yes/no  measurable  precipitation  forecasts  for  12 
hour  periods  at  60  stations.  The  accuracy  of  these  forecasts  is  measured  in  terms 
of  threat  score.  Annually  averaged  threat  scores  for  several  projections  are  shown 
in  Figure  7.  The  annual  threat  scores  produced  by  the  numerical  forecast  models 
are  shown  for  comparison.  The  forecast  projections  for  the  models  are  from  model 
initial  time;  projections  for  the  subjective  forecasts  are  from  issue  time.  Over 
the  period  of  record  both  sets  of  forecasts  improved.  On  the  basis  of  linear 
regression,  the  10  year  increases  in  skill  score  of  the  36  hour  surface  and  500 
millibar  objective  prognosis  were  11.5%  and  13.5%  respectively.  The  10  year 
percentage  increases  of  threat  score  for  the  NMC  guidance  forecasts  were  24X  for  the 
shorter  projection  and  15%  for  the  longer.  The  subjective  forecasts  use  the 
objective  forecasts  as  guidance.  Since  the  trends  are  similar  (correlation  about 
.7),  it  appears  that  since  1970  the  increasing  skill  of  the  subjective  forecasts  is 
mostly  due  to  improvements  in  the  objective  guidance. 

Smoothed  annual  percent  correct  of  local  yes/no  precipitation  forecasts  are  shown 
in  Figure  8.  Data  are  available  for  three  cities  (Washington,  Boston,  and  Chicago) 
since  1954  and  for  50  forecast  offices  since  1966.  The  regression  line  for  the 
longer  record  shows  an  average  rate  of  increase  of  2.1%  per  10  year  interval.  The 
trend  for  the  50  WSFOs  is  not  significant  at  the  95%  level.  The  improvement  was 
not  steady;  there  was  negligible  improvement  from  the  mid  1960s  to  the  mid  1970s. 

The  sharp  rise  after  1975  is  encouraging,  but  more  data  are  needed  to  establish 
authenticity.  We  cee  that  even  though  the  skill  of  the  guidance  forecasts  increased 
during  the  early  1970s  along  with  the  improvements  of  the  36  hour  surface  and  500 
millibar  prognoses,  the  improvements  in  the  local  forecasts  issued  to  the  public 
were  smaller.  This  result  was  not  true,  however,  for  the  probability  of  measureable 
precipitation  forecasts. 

Probability  Forecasts  of  Measurable  Precipitation 

Probability  forecasts  of  precipitation  are  issued  to  the  public  by  the  WSFOs  for 
the  periods  3-15  hours,  15  -  27  hours,  and  27  -  39  hours  after  issuance  (today, 
tonight,  and  tomorrow) .  Graphs  of  the  annual  verification  for  about  50  WSFOs  are 
shown  in  Figure  9.  The  dashed  line  represents  automated  guidance  (MOS  since  1970) 
issued  four  hours  earlier.  The  general  level  of  performance  for  the  three  periods 
respectively  were  35,  24,  and  15  percent  improvement  of  Brier  Score  relative  to 
climatology.  The  WSFO  forecasts  indicate  an  improving  trend  with  most  of  the 
improvement  occurring  after  1971.  The  average  10  year  percentage  increase  in  skill 
score  was  about  6%,  36%,  and  66%  in  the  first,  second,  and  third  periods  respectively. 
The  trend  is  not  statistically  significant  for  the  first  period,  but  is  highly 
significant  for  the  second  and  third  periods.  The  10  year  increases  in  skill  score 
were  6.9%  and  7.4%  respectively  for  the  second  and  third  period  forecasts,  about 
half  that  of  large  scale  circulation  forecasts  of  about  the  same  projection  (Charba 
and  Klein,  1980b) . 

Since  1975  the  subjective  forecasts  have  been  highly  correlated  with  the  MOS  fore- 
costs.  There  is  a  slight  indication  of  MOS  gaining  on  the  forecasters  in  the  first 
period,  and  the  forecasters  gaining  on  MOS  in  the  second  and  third  periods.  Further 
study  of  these  data  show  very  significant  differences  in  the  trends  of  cool  and 
warm  season  probability  of  precipitation  forecasts  (Figure  10).  Improvement  was 
greater  in  the  cool  season  when  most  precipitation  is  caused  by  the  synoptic  scale 
systems  which  the  numerical  models  are  skillful  in  forecasting. 
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Quantitative  Forecasts  of  Precipitation  Amount 

Specialized  meteorologists  at  the  National  Meteorological  Center  have  been  preparing 
quantitative  precipitation  forecasts  for  the  nation  since  1961.  Verification  data 
are  available  for  amounts  equal  or  greater  than  0.5  inch  and  1.0  inch  for  the 
periods  0  to  24  hours  and  24  to  48  hours.  The  trend  for  the  0  to  24  hour  forecasts 
was  level  up  through  1971.  From  1972  to  1979  the  10  year  change  in  threat  score 
was  about  .08  for  both  amounts  and  both  periods  —  about  the  same  improvement  as 
the  NMC  subjective  categorical  precipitation  forecasts.  The  centralized  forecasts 
for  heavy  snow  show  a  small  upward  trend  over  the  past  18  years  (Charba  and  Klein, 
1980b) .  Zuckerberg  (1980)  found  a  small  long  term  skill  increase  in  forecasts  of 
heavy  snow  events  for  nine  eastern  cities. 

Other  National  Weather  Service  Forecasts 

The  Techniques  Development  Laboratory  conducted  a  verification  program  to  help 
validate  and  to  improve  use  of  the  operational  Model  Output  Statistics  (MOS) 
forecasts.  These  data  are  summarized  by  season  for  the  past  four  to  five  years  in 
a  series  of  Office  Notes.  Although  the  period  of  record  is  short,  some  trends  are 
indicated.  He  have  used  data  from  only  one  of  these  reports  for  illustration 
(Hebenstreit,  et  al,  1979).  The  data  are  presented  in  terms  of  early  and  find 
guidance  issued  about  3  and  6  hours  after  initial  map  time,  and  local  and  per¬ 
sistence  forecasts  issued  about  9  to  10  hours  after  initial  map  time.  All  forecasts 
use  the  latest  available  observation,  and  the  local  forecast  uses  the  guidance 
as  well. 

Surface  Wind 


Figure  11  shows  verification  data  for  surface  wind  direction  in  terms  of  mean 
absolute  error.  In  general  there  has  been  steady  improvement,  paralleling  improve¬ 
ment  in  the  MOS  guidance.  The  surface  windspeed  verification  data  (not  shown) 
provided  similar  findings,  but  there  was  negligible  improvement.  Two  category 
verification  (LTE  22  kts  and  GT  22  kts)  is  shown  in  Figure  12.  The  local  forecasts 
show  no  significant  trend  although  the  MOS  forecasts  show  improvement  after  1974, 
especially  for  the  42  hour  forecasts. 

Sky  Cover 

The  skill  score  for  sky  cover  forecasts  is  shown  in  Figure  13.  There  has  been  a 
4-51  improvement  by  the  local  forecasters  over  the  five  winter  seasons.  However, 
the  18  hour  objective  forecast  (MOS)  is  superior  to  the  locals.  The  42  hour  MOS 
forecasts  have  been  more  than  twenty  five  percent  better  than  the  local  forecasts 
for  the  entire  period,  and  show  a  good  upward  trend. 

Ceiling 

Ceiling  forecast  verification  for  four  seasons  is  shown  in  Figure  14.  The  12Z  (HR) 
and  15Z  (HR)  local  forecasts  have  less  skill  than  persistence.  These  forecasts 
are  3  and  6  hour  forecasts  from  time  of  latest  observation  for  persistence  and 
local  forecasts]  projections  for  the  objective  forecasts  are  about  six  hours 
longer.  There  was  no  significant  trend.  The  MOS  forecasts  show  mixed  results. 
Longer  range  forecast  verification  is  shown  in  Figure  15.  Local  skill  is  small  and 
is  less  than  persistence.  The  MOS  forecasts  are  better  than  either,  and  indicate 
an  improving  trend.  However,  this  trend  parallels  the  improvement  of  persistence 
and  might  not  be  significant. 

visibility 

Verification  data  for  visibility  forecasts  are  shown  in  Figures  16  and  17.  The  12Z 
(HR)  local  forecasts  are  worse  than  persistence,  and  there  is  no  trend.  The  local 
forecasts  improved  over  persistence  for  the  21Z  (HR)  verification  time,  These 
forecasts  are  for  the  same  projections  as  the  ceiling  forecasts. 
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Temperature  Forecasts 

Verification  data  for  maximum  and  minimum  temperatures  are  shown  in  Figure  18  and 
19  respectively.  The  local  forecasters  achieved  slight  improvement  for  the  24  hour 
maximum  temperature  and  moderate  improvement  for  the  48  hour  maximum  temperature 
forecasts.  The  latter,  however,  closely  parallels  improvements  of  MOS.  No  sig¬ 
nificant  trend  is  evident  for  the  minimum  temperature  forecasts  except  that  the  MOS 
improved  in  1975  to  be  comparable  to  the  local  forecasters.  Note  that  the  local 
forecasters  have  about  six  additional  hours  of  data  and  access  to  the  MOS  early 
forecasts. 

Severe  Storm  Forecasts 


An  analysis  of  forecast  skill  at  the  National  Severe  Storms  Forecast  Center  was 
presented  by  Pearson  and  Weiss  (1979) .  The  skill  level  (critical  success  index) 
of  the  severe  local  storm  outlook  has  been  increasing  slowly  (Figure  20) .  The 
increase  is  attributed  to  the  improvement  of  the  LFM  model.  The  trend  for  tornado 
watches  (Figure  21)  suggests  a  slight  improvement.  These  results  were  attributed 
to  the  difficulties  inherent  in  attempting  to  make  mesoscale  forecasts  using 
synoptic  scale  observation  networks  and  forecast  models.  Radar  and  satellite  data 
are  mainly  useful  for  diagnostic  activities. 

The  Western  Region  Program 

The  MOS  forecasts  of  probability  of  precipitation  became  available  to  National 
Weather  Service  Western  Region  forecasters  in  1972.  In  the  following  three  years 
the  MOS  forecasts  improved  significantly,  while  the  manual  forecasts  improved 
little  (Figure  22).  The  Western  Region  evaluated  these  results,  and  determined 
that  it  should  be  possible  for  forecasters  to  improve  the  MOS  forecasts.  A  motiva¬ 
tional  program  was  implemented  1  Dec  75  (MacDonald,  1977;  Snellman,  1977)  in 
which  management  stressed  improvement  over  MOS.  The  Region  Headquarters  provided 
weekly  verification  feedback  and  station  ranking.  Technical  information  also  was 
produced  and  crossfed.  This  program  promoted  forecaster  interest  in  MOS,  in  using 
their  individual  forecasting  skills,  and  in  sharpening  their  concern  for  the 
quality  of  the  forecast  product.  The  results  were  that  forecasters  were  motivated, 
became  better  meteorologists,  and  produced  the  best  probability  of  precipitation 
forecasts  ever  issued  in  the  Western  Region. 

Ceiling  and  visibility  forecasts  were  also  evaluated  in  this  program.  Figure  23 
shows  Snellman' s  verification  contingency  tables.  The  overall  percent  correct  for 
ceiling  and  for  visibility  forecasts  were  practically  the  same  for  both  the  MOS  and 
the  Western  Region  forecasters.  However,  the  forecasters  percent  correct  for  the 
lowest  three  categories  (less  than  1000  feet  and  3  miles)  was  much  higher  (28%  and 
33%)  than  MOS  (11%  and  13%).  Snellman  also  pointed  out  that  the  forecasters  had  a 
much  better  bias  than  MOS.  However,  the  MOS  bias  has  been  improved  in  recent 
years. 

Air  Weather  Service  Ceiling/Visibility  Forecasts 

The  Air  Waather  Service  quarter  month  verifications  for  all  hours  and  for  3,  6,  12, 
and  24  hours  are  shown  in  Figures  24  -  28  respectively  in  terms  of  percentage 
improvement  over  persistence.  Linear  regression  lines  are  shown  dashed.  The  10 
year  improvements  of  skill  score  are  roughly  12.5,  25.1,  16.0,  6.8,  and  8.4  percent 
for  all,  3,  6,  12,  and  24  hours  respectively.  The  large  improvement  is  due  partly 
to  the  initial  low  skill.  This  performance  still  leaves  much  to  be  desired.  The 
forecasts  are  biased  toward  forecasts  of  the  highest  categories.  In  addition,  the 
skill  of  forecasts  for  the  lowest  category  is  worse  than  persistence  for  forecasts 
of  six  hours  or  longer.  Most  of  the  improvement  was  in  the  higher  categories;  only 
small  skill  was  obtained  for  ceilings  and  visibilities  less  than  1000  feet  and/or 
two  miles. 

Forecasting  in  Great  Britain  (Mason,  1980) 

Since  1963  the  0755L  forecast  valid  for  the  following  16  hours  and  the  1755L 
forecast  valid  for  the  following  30  hours  have  been  subjectively  evaluated.  The 
elements  (wind  speed  and  direction,  weather,  state  of  sky,  and  maximum  temperature) 
are  each  awarded  marks  2,  1,  or  0  according  to  whether  the  forecast  is  correct, 
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partly  correct  or  incorrect.  The  scores  are  similar  for  the  four  elements,  and  an 
average  of  the  marks  is  taken  as  a  measure  of  the  overall  performance.  These 
forecasts  are  based  on  guidance  from  the  Bracknell  Central  Forecastinq  Office 
issued  some  2  1/2  hours  earlier,  in  turn  depending  heavily  on  the  predictions  of 
the  numerical  models  which  have  shown  marked  improvement  in  recent  years.  Ten 
years  ago  less  than  half  of  these  forecasts  were  essentially  correct,  and  1  in  7 
were  seriously  misleading.  Since  1972  two-thirds  have  been  essentially  correct, 
and  only  1  in  15  has  contained  serious  errors.  Significant  improvements  were  also 
noted  for  the  48  hour  and  72  hour  projections. 

Discussion 


In  general,  most  of  the  forecasts  produced  by  the  numerical  weather  prediction 
models,  including  Model  Output  Statistics,  have  shown  improvement  over  the  periods 
for  which  data  were  available.  The  significance  of  each  improvement  is  difficult 
to  assess  because  of  the  variety  of  scoring  techniques  and  the  time  periods. 

However,  rough  comparisons  were  possible  for  the  elements  verified  in  terms  of  a 
skill  score.  A  skill  score  for  the  numerical  models  (SS  -  2  (70  -  SI)))  was 
developed  by  Shuman  (1978).  On  the  basis  of  linear  regression  lines,  the  10  year 
rate  of  increase  of  skill  score  of  the  primitive  equation  (PE)  36  hour  sea  level 
and  500  millibar  forecasts  were  11.5%  and  13  51  respectively  (Charba  and  Kl«*in, 
1980b) .  These  regression  lines  were  based  on  12  years  of  data  except  that  the  500 
millibar  values  included  an  additional  five  years  of  baroclinic  model  performance. 

In  general,  forecasting  skill  decreases  with  increasing  projection  time;  however, 
the  rate  of  improvement  in  skill  generally  is  largest  for  forecasts  of  36  to  48 
hour  projection.  As  a  conservative  rough  estimate,  we  use  a  10  year  rate  of  improve¬ 
ment  of  10%  as  roughly  comparable  to  the  improvement  rate  obtained  by  the  numerical 
models.  The  following  types  of  forecasts  seem  to  be  meeting  or  exceeding  this 
proposed  standard. 

a.  SUNYA  consensus  24  and  48  hour  precipitation  percent  improvement  over 
climatology  (Figure  5). 

b.  NWS  locally  issued  and  MOS  probability  of  precipitation  forecasts  for 
second  and  third  period  for  50  weather  service  forecast  offices  (WSFO)  (Figure  9) . 
Most  of  this  improvement  is  for  the  cool  season  forecasts  (Figure  10) . 

c.  MOS  categorical  wind  speed  for  both  the  18  and  42  hour  projections  (Figure 
12).  Only  six  data  points  are  available;  so  the  significance  is  uncertain. 

d.  MOS  sky  cover  18  and  42  hour  forecasts  (Figure  13) .  Only  5  data  points. 

e.  MOS  18  hour  ceiling  forecasts.  Only  four  data  points.  Also  persistence 
scored  high  (Figure  15) . 

f.  MOS  18  hour  visibility  forecasts.  Only  four  data  points  (Figure  17). 

g.  AWS  3,  6,  12,  and  24  hour  four  category  ceiling/vieibility  forecasts. 
Significant  at  the  95%  level  (Figures  25  through  28). 

Note  that  the  above  list  mostly  consists  of  MOS  forecasts;  exceptions  are  SUNYA 
probability  of  precipitation  (POP),  NWS  POP,  and  AWS  ceiling/  visibility  forecasts. 

Other  types  of  forecasts  which  appear  to  have  a  similar  improving  trend,  but  not 
directly  comparable  include: 

a.  NMC  subjective  and  objective  categorical  forecasts  of  measurable  precipita¬ 
tion,  first  and  second  periods  (Figure  7) . 

b.  Washington,  Boston,  and  Chicago  subjective  categorical  forecasts  o"  measur¬ 
able  precipitation  (Figure  8). 

c.  MOS  and  local  surface  wind  direction  forecasts.  Six  data  points  (Figure 

11). 
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d.  MOS  and  local  maximum  temperature  forecasts  (Figure  18) . 

e.  NWS  severe  weather  outlook  forecasts.  (Figure  20). 

f.  Centralized  quantitative  precipitation  forecasts  of  1/2  inch  and  1  inch  and 
heavy  snow  (Charba  and  Klein,  1980b) . 

The  remaining  types  of  forecast  show  little  or  small  improvement.  These  include: 

a.  MIT  consensus  probability  forecasts  for  minimum  temperature  and  class  of 
precipitation  amount  for  four  projections  (Figures  2  and  3) . 

b.  SUNYA  consensus  forecasts  for  temperature  for  four  projections  and  precipit 
ation  for  the  72  and  96  hour  projections  (Figures  4  and  5) . 

c.  Categorical  forecasts  for  measurable  precipitation  issued  locally  by  50 
WSFOs  (Figure  8) . 

d.  Warm  season  probability  of  measurable  precipitation  (Figure  10) . 

e.  MOS  and  local  wind  speed  (Hebenstreit,  et  al;  1979) . 

f.  Categorical  local  surface  wind  speed  forecasts  (Figure  12). 

g.  Local  sky  cover  forecasts  (Figure  13) . 

h.  Local  and  MOS  12  -  15  hour  ceiling  forecasts  (Figure  14),  and  local  21  hour 
ceiling  forecasts  (Figure  15) . 

i.  Local  and  MOS  12  -  15  hour  visibility  forecasts  (Figure  16)  and  local  21 
hour  visibility  forecasts  (Figure  17) . 

j.  Local  minimum  temperature  forecasts  (Figure  19). 

k.  NWS  tornado  watch  forecasts  (Figure  21) . 

We  conclude  that  significant  improvement  has  been  achieved  in  forecasting  nearly 
every  element  by  at  least  one  group  of  forecasters.  This  is  contrary  to  the  often 
made  assertion  of  lack  of  progress.  We  note  that  progress  obtained  by  one  group  or 
method  often  does  not  seem  to  be  successfully  attained  by  another  group  making 
similar  forecasts.  The  objective  forecasts,  especially  MOS,  seem  to  offer  the  best 
chances  of  effecting  improvement  to  all  consumers  for  the  12  to  96  hour  projections 
for  those  elements  in  which  the  objective  skill  roughly  equals  manual  skill.  We 
need  continued  effort  to  improve  MOS,  to  make  forecasters  aware  of  the  quality  of 
the  guidance  and  to  encourage  them  to  use  it.  This  process  seems  to  be  occurring 
across  the  Nation.  Still,  it  is  curious  that  not  all  methods  led  to  similar  per¬ 
formance  improvements  for  similar  forecast  elements  and  projections.  Since  all 
forecasting  groups  in  the  CONUS  use  the  NWS  facsimile  package  as  the  primary 
guidance,  we  infer  that  factors  other  than  the  quality  of  the  guidance  product  must 
be  important  in  determining  the  performance  achieved.  Some  of  these  factors  are 
discussed  in  the  next  section. 


FACTORS  AFFECTING  FORECASTING  PERFORMANCE 

Significant  improvements  in  weather  forecasting  have  been  achieved  in  recent  years; 
however,  the  improvements  were  not  achieved  for  every  element  by  every  forecasting 
group.  Since  the  skill  of  the  numerical  weather  prediction  model  guidance  products 
has  steadily  improved  in  accuracy,  factors  other  than  the  quality  of  the  guidance 
apparently  are  important  in  determining  the  quality  of  the  forecasts  provided  to 
customers.  Over  the  years,  small  improvement  in  weather  forecasting  has  been  a 
recurrent  theme  in  meteorological  literature;  suggestions  given  for  improving  fore¬ 
casting  included  meteorological  education,  forecasting  experience,  individual 
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aptitude  and  interest,  additional  real  time  observations,  bigger  and  faster  computers, 
better  numerical  models,  and  improvement  of  weather  station  operations.  These 
suggestions,  except  for  those  addressed  in  preceding  sections  (observations,  computers, 
models) ,  will  be  discussed  as  factors  that  potentially  affect  the  rate  of  improvement 
of  forecasting  skill. 

Meteorological  Education 

Many  consider  university  education  or  its  equivalent  mandatory  for  meteorologists 
(Baum,  1975;  Newton,  1980).  The  World  Meteorological  Organization  recommended 
training  program  for  forecasters  (Class  II  and  Class  I  meteorologists)  is  equivalent 
to  university  education  (see  description  by  Van  Mieghem,  1963).  In  most  countries, 
university  meteorological  education  is  required  for  meteorologists;  the  only  excep¬ 
tions  are  the  U.S.  military  enlisted  forecaster  and  the  occasional  practice  of 
accepting  experience  in  lieu  of  formal  education.  Jenkins  (1953)  evaluated  92 
forecasters  ranked  in  the  Air  Weather  Service  (AWS)  forecasting  skill  verification 
program  (AWS,  1943).  No  significant  differences  were  found  between  better  and  poorer 
forecasters  with  respect  to  education,  college  major,  or  mathematics  background. 
Kershner  (1957)  used  the  peer  evaluation  technique  to  identify  a  number  of  good  and 
poor  Air  Weather  Service  forecasters.  Subsequently  Kershner  and  Jenner  (1959)  and 
Kershner  (1962)  analyzed  the  characteristics  of  300  officers  and  129  warrant  officers/ 
enlisted  weather  forecasters.  They  found  that  the  good  forecasters  in  each  group 
had  more  education,  especially  in  mathematics  and  physics,  than  the  poorer  fore¬ 
casters.  No  comparisons  were  reported  between  officers  and  enlisted. 

In  a  university  game  at  City  College  of  New  York,  Gedzelman  (1978)  found  no  sig¬ 
nificant  difference  in  the  forecasting  ability  of  meteorologically  educated  and 
uneducated  forecasters  (sample  of  12).  These  results  are  supported  by  Bosart  (1975) 
and  Sanders  (1979) .  Each  started  with  people  who  were  meteorologically  uneducated. 

The  programs  began  with  a  short  course  taught  concurrently  with  the  conduct  of  the 
forecasting  exercise.  After  the  first  couple  months,  individual  skill  became  level. 

The  excellent  performance  results  achieved  by  AWS  forecasters  (Fifth  Weather  Wing, 

1980;  German  and  Hicks,  1980)  indicate  that  the  18  week  weather  forecaster  training 
conducted  by  the  Chanute  Technical  Training  Center  also  provides  enough  knowledge 
of  hasic  meteorological  processes  to  do  good  weather  forecasting.  This  course 
alt  reaches  a  number  of  Air  Force  specific  operational  procedures  that  shorten 
subsequent  weather  station  orientation;  thus,  it  should  not  be  shortened.  Cahir 
(1978)  described  experiments  with  an  interactive  system  in  which  he  indicated 
that  the  system  is  used  successfully  only  by  those  who  are  able  to  integrate 
it  into  a  good  scientific  approach.  This  implies  a  need  for  more  than  the 
elementary  education  mentioned  above;  however,  no  supporting  data  were  presented. 


We  have  noted  that  over  the  last  ten  years  the  Air  Weather  Service  forecasters  have 
become  increasingly  less  meteorologically  educated,  but  the  overall  forecasting 
performance  apparently  has  not  deteriorated.  One  of  the  reasons  is  that  meteoro¬ 
logically  educated  personnel  monitor  the  performance  and  provide  guidance  on  develop¬ 
ment  of  logical  scientific  approaches  to  forecasting.  In  addition,  guidance  is 
provided  on  the  strengths  and  weaknesses  of  the  objective  and  subjective  forecasting 
aids.  Use  of  guidance  such  as  the  satellite  discussion  bulletins  and  map  discussion 
bulletins  prepared  by  meteorologically  educated  personnel  is  encouraged.  Solutions 
to  forecast  problems  are  documented  through  case  studies.  Formal  publications, 
refresher  training  of  basic  concepts,  and  an  active  staff  visit  program  also  help. 

University  education  apparently  provides  a  breadth  and  depth  of  knowledge  not  needed 
for  routine  base  weather  station  forecasting.  Even  though  this  knowledge  is  not 
required  for  hour-by-hour,  day-by-day  forecasting,  it  is  useful  in  complex  fore¬ 
casting  situations,  solving  forecasting  problems,  and  consulting.  For  example:  Some 
(  units  were  having  a  problem  with  forecasting  freezing  precipitation  in  air  masses 

i  that  were  colder  than  freezing.  Thorough  study  of  meteorological  literature  by  a 

j  meteorologist  revealed  that  cold  stratus  clouds  often  produce  freezing  drizzle. 

I  Several  conditions  needed  for  drizzle  to  occur  were  identified.  The  physical 

basis  for  the  problem  was  identified,  and  guidance  was  given  for  forecasting.  In 
this  instance  the  National  Weather  Service  subsequently  verified  the  result  for  all 
CONUS  (Bocchisrri,  1979) .  Such  efforts  require  s  university  educated  meteorologist. 
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Although  the  general  consensus  of  professional  meteorologists  is  that  university 
education  is  required  to  make  an  individual  a  good  weather  forecaster,  avi.ilable 
data  generally  do  not  provide  support.  Kershner  and  Jenner  found  that  good  fore¬ 
casters  tended  to  have  more  university  education  and  poor  forecasters  tended  to  have 
less,  but  Jenkins  found  no  relationship  between  forecasting  skill  and  education. 

The  skill  demonstrated  by  the  AWS  forecasters,  now  mostly  having  no  university 
education  clearly  demonstrates  that  university  education  is  not  required.  Chanute 
training  is  adequate  for  good  quality  terminal  weather  forecasting.  We  have  noted, 
however,  that  for  continued  good  performance  the  forecasting  system  must  have 
university  educated  personnel  within  it  to  help  with  problems,  planning,  training, 
and  application  of  new  technology. 

Forecasting  Experience 

Forecasting  experience  (total  and  on  station)  intutitively  is  highly  desirable  for 
weather  forecasters.  Early  in  this  century,  experience  was  a  primary  tool  used  by 
forecasters;  it  continues  to  be  considered  important  in  1980.  It  is  logical  to 
conclude  that  experience  is  a  major  factor  since  most  good  forecasters  have  expe¬ 
rience;  however,  we  acknowledge  that  there  probably  are  more  experienced  forecasters 
of  moderate  skill  than  of  good  skill.  Jenkins  (1953),  Kershner  (1957),  Gedzelman 
(1978),  Bosart  (1975),  and  Sanders  (1979)  indicated  that  forecasting  experience  has 
little  relationship  to  forecasting  performance.  In  the  latter  three  studies, 
neophyte  forecasters  became  competitive  with  the  more  experienced  forecasters  within 
one  to  two  months  of  elementary  meteorological  education  and  practice.  The  Air 
Weather  Service  forecasting  performance  (German  and  Hicks,  i960),  which  showed 
significant  and  increasing  improvement  over  persistence,  from  1968  through  1979,  was 
achieved  by  a  force  in  which  many  of  the  forecasters  had  less  than  three  years 
experience,  especially  toward  the  end  of  the  period.  Lininger  (1979)  reported  on  a 
two  year  project  in  which  forecasters  in  his  office  participated  in  making  3  and  24 
hour  ceiling  and  visibility  forecasts  using  very  limited  data  (one  surface  and  500 
mb  map  on  Mondays,  and  the  local  official  observation  each  day) .  These  forecasters 
were  university  educated  meteorologists,  but  not  experienced  in  the  area.  The 
forecasts  produced  were  skillful  relative  to  climatology;  forecasters  achieved 
skill  in  one  to  two  months,  with  little  improvement  thereafter. 

The  minimum  thresholds  of  forecasting  experience  seem  to  be  of  the  order  of  a  couple 
of  months,  not  years,  for  point  (terminal)  forecasts.  This  conclusion  is  supported 
by  data  analyses  made  by  several  individuals;  however,  there  undoubtedly  are  excep¬ 
tions.  Forecasting  experience  apparently  gives  an  individual  a  better  sense  of  the 
spectrum  of  weather  that  could  occur  for  a  given  synoptic  pattern,  but  this  sense, 
or  knowledge,  doesn't  necessarily  become  better  forecasting.  Perhaps  these  indi¬ 
viduals  are  not  gifted  in  accurately  selecting  the  most  provable  weather,  or  these 
instances  occur  too  infrequently  to  show  up  in  the  mass  of  verification  data. 
Gedzelman  (1978)  presented  one  example  in  which  an  experienced  forecaster  correctly 
forecast  a  backdoor  cold  frontal  passage  not  forecast  by  guidance.  Even  though  the 
possibility  was  discussed  by  all  forecasters  the  inexperienced  forecasters  followed 
the  machine  objective  guidance.  Kershner  and  Jenner  (1959)  found  that  the  age  at 
which  Air  Force  forecasters  began  forecasting  was  important;  those  who  began  before 
age  25  were  better. 

A  few  other  specific  instances  of  positive  benefit  of  experience  are  available. 
Phillips  (1979)  cited  an  instance  in  which  use  of  a  case  study  resulted  in  an 
accurate  forecast  of  a  rare  event.  George  (1960)  presented  numerous  forecasting 
aids  developed  partly  through  forecaster's  experience.  German  (1979b)  cited  a  case 
study  of  a  forecast  at  Mountain  Home  AFB,  Idaho  that  was  successful  in  large  part 
because  of  the  forecaster's  experience  with  similar  situations.  Safford  and  Gesser 
(1977)  described  a  procedure  for  forecasting  passage  of  lake  effect  snow  showers 
over  airfields.  Note  that  these  examples  all  are  of  documented  experience.  Un¬ 
doubtedly,  there  are  numerous  other  examples.  By  the  time  a  station  has  existed 
for  about  five  years,  most  of  the  important  synoptic  and  local  effects  should  have 
been  identified  and  documented  in  a  station  technical  reference  document  or  in  case 
studies.  IT  such  documentation  is  not  done,  then  each  individual's  insights  un¬ 
necessarily  depart  upon  reassignment  of  the  individual.  Experience  apparently  is 
helpful  when  it  includes  thorough  post  analysis  and  learning.  In  such  cases,  learn¬ 
ing,  not  experience,  probably  is  the  effective  factor,  and  could  just  as  easily  be 
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done  before  as  after  the  fact.  This  dees  not  mean  more  education,  only  thorough 
review  of  available  station  references,  climatology,  case  studies,  etc.,  seasonally 
or  more  often  as  required.  If  this  concept  is  accepted,  it  would  be  logical  to 
include  time  for  this  learning  in  the  station  manpower  computation. 


Overall  we  must  conclude  that  forecasting  experience  is  only  a  minor  factor  in 
terminal  weather  forecasting  performance  at  weather  stations.  It  may  have  been 
important  in  the  past,  and  it  may  still  be  important  where  the  necessary  information 
is  not  available  to  learn.  Documentation  of  forecasting  experience  in  an  organized, 
systematic  way  for  forecasters  to  learn  is  more  important.  This  should  be  con¬ 
sidered  and,  if  possible,  provided  for  when  weather  stations  are  modernized. 


Aptitude  and  Interest  in  Weather  Forecasting 

Willett  (1951)  stated  that,  "The  empirical  and  esoteric  nature  of  most  weather 
forecasting  today  places  a  high  premium  on  the  natural  aptitude  of  the  individual 
forecaster.  This  natural  aptitude  is  invariably  indicated  by  a  strong  liking  for 
and  sustained  interest  in  weather  forecasting^for  its  own  sake  over  a  long  period  of 
time...."  Willett  went  on  to  say  that  aptitude  and  interest  were  not  necessarily 
correlated  with  meteorological  education.  Aptitude  and  interest  are  not  prerequisites 
for  employment  as  a  weather  forecaster  today.  Military  forecasters  for  example, 
are  assigned  forecasting  duties  on  the  basis  of  completion  of  an  education  program. 
Aptitude  and  interest  apparently  are  developed  serendipitiously  at  various  stages  of 
a  forecaster's  career.  However,  some  management  actions  appear  to  have  the  effect 
of  decreasing  enthusiasm  for  exercising  aptitude  and  interest.  Actions  needed  or 
taken  to  correct  this  trend  have  been  discussed  by  Snellman  (1977) ,  Kaehn  (1978)  , 
and  Schwartz  (1980) . 


The  studies  reported  by  Kershner  (1957),  Sanders  (1973  and  1979),  Bosart  (1975),  and 
Gedzelman  (1978)  indicate  aptitude  and  interest  are  important  factors  which  con¬ 
tribute  toward  good  forecasting  performance.  Ramage  (1976),  in  assessing  Sanders 
(1973)  findings  commented,  "Perhaps  this  can  be  explained  in  terms  of  the  innate 
ability  of  some  individuals  to  think  in  a  strongly  nonlinear  way  and  so  to  antic¬ 
ipate  development  . . .  from  insignificant  beginnings  more  accurately  than  their 
colleagues."  AWS  management  has  observed  that  people  with  an  intense  interest  in 
weather  forecasting  usually  provide  good  forecasting  service.  In  the  military  this 
often  is  recognized  by  the  customer;  in  some  cases  senior  officers  seek  advice  from 
the  forecaster  who  demonstrates  the  most  interest  and  aptitude  in  forecasting. 
Kershner  (1967)  found  five  tests  that  correlated  well  with  forecasting  skill; 
however,  these  tests  were  never  used  operationally  to  select  weather  forecaster 
candidates.  There  is  little  data  either  supporting  or  not  supporting  the  contention 
that  forecaster  aptitude  and  interest  are  important.  It  is  not  known  whether  these 
characteristics  are  inherent  in  good  forecasters  or  are  brought  out  by  good  managers. 

Weather  Station  Operations  (Systematic  Forecasting  Procedures) 

Improvement  of  station  operations  was  addressed  by  Willett  (1951),  and  has  been 
discussed  occasionally  (Fulks,  1940;  Snellman,  1975,  Kaehn,  1978;  German,  1979a; 
Schwartz,  1980  for  example).  Willett  recognized  that  the  abundance  of  analysis  and 
forecasting  tools  produced  during  the  1940s  sometimes  had  an  adverse  impact  in 
weather  station  operations.  Various  forecasters  or  forecasting  teams  often  used 
differing  approaches  and  tools,  resulting  in  poor  continuity.  Since  the  main 
forecasting  method  was  extrapolation,  the  results  could  be  disasterous.  He  advocated 
adoption  of  one  set  of  analyses  and  procedures  to  be  used  by  all  forecasters,  not 
only  within  a  station  but  throughout  a  service.  Standardization  of  weather  fore¬ 
casting  activities  has  been  advocated  for  at  least  30  years.  Little  progress  has 
been  made  except  where  manually  produced  forecaster  aids  have  come  to  be  produced  by 
computer.  Even  then  the  role  of  the  person  in  the  forecasting  process  is  handled 
differently  nearly  everywhere. 

Forecasters  build  their  forecasts  subjectively  using  whatever  is  available  and  can 
be  assimilated  in  the  available  time.  While  forecasting  no  weather  is  done  correctly 
about  90  percent  of  the  time,  forecasting  significant  weather  needs  improvement. 

Much  of  the  important  weather  is  associated  with  structures  which  appear  to  be 
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synoptically  weak,  but  in  reality  may  be  strong  in  the  mesoscale.  Ways  are  needed 
to  identify  these  structures  and  to  forecast  them.  In  addition  improvement  is  needed 
in  forecasting  cloud  formation,  decay,  and  translation.  Some  of  these  problems  could 
be  mitigated  with  improved  synoptic  scale  models,  but  most  are  likely  to  need  a  much 
finer  scale  approach.  Today's  mesoscale  models  may  be  a  good  approach,  but  are  not 
likely  to  be  of  significant  help  within  the  next  10  years  due  to  the  state  of  the 
art  and  the  impracticality  of  running  them  for  every  location.  Thus,  for  the  next 
few  years  at  least,  efforts  to  develop  systematic  approaches  to  forecasting  will 
need  considerable  effort. 

There  are  several  programs  working  toward  automation  of  large  parts  of  station 
procedures  (Naval  Environmental  Display  Station  (NEDS)  -  Thormeyer,  1978;  Automation 
of  Field  Operations  and  Services  (AFOS)  -  Klein,  1978;  Prototype  Regional  Observation 
and  Forecasting  System  (PROFS)  -  Beran  an3  Little,  1978,  and  Automated  Weather  Display 
System  (AWDS)  -  Best,  1977).  In  addition  the  roles  of  the  man  and  the  machine  were 
discussed  at  the  1978  AMS  Conference  on  Weather  Forecasting  and  Analysis  (Snellman  and 
Murohv,  1978) .  Overall  the  consensus  is  that  weather  forecasting  will  improve  with  the 
implementation  of  these  systems,  but  there  are  no  studies  with  data  illustrating 
that  improvement  will  result  from  the  actions  now  being  implemented.  None  of  these 
systems  can  run  mesoscale  models  of  sufficient  sophistication  to  be  helpful. 

Another  factor  that  often  is  mentioned  as  important  is  the  station  workload.  Work¬ 
load  in  weather  stations  has  changed  significantly.  Thirty  years  ago  most  forecast 
offices  were  24  hour  stations  and  forecasters  spent  high  percentages  of  their  time 
in  meteorological  analysis,  prognosis,  and  forecasting.  Now  most  of  these  synoptic 
meteorological  analysis  and  prognosis  tasks  are  done  in  weather  centrals.  The  time 
saved  was  intended  to  increase  the  time  available  for  the  forecaster  to  think 
through  the  meteorology  to  develop  more  accurate  forecasts.  Instead,  forecasters 
have  been  assigned  other  duties  such  as  briefing  and  coordinating.  In  addition  many 
weather  stations,  especially  military,  are  only  open  part  time.  The  workload  is 
added  generally  because  of  reductions  in  forces  due  to  rapid  rises  of  personnel 
costs,  and  because  significant  weather  tends  to  occur  relatively  infrequently.  In 
the  predominantly,  not-hazardous  weather  situations  the  forecaster  is  likely  to  be 
perceived  as  being  able  to  handle  additional  tasks,  with  the  result  that  management 
determines  that  fewer  people  are  required  and  assigned.  The  development  of  devices 
to  assist  forecasters,  such  as  Automation  of  Field  Operations  and  Service,  Automated 
Weather  Distribution  System,  and  Naval  Environmental  Display  Station  have  not  been 
heralded  as  having  significant  impacts  on  the  long  term  trends  noted.  Overall,  the 
amount  of  time  forecasters  have  available  to  diagnose  and  forecast  probably  has  not 
increased,  and,  except  where  forecasters  have  developed  an  attitude  and  interest  to 
forecast  well,  there  are  indications  that  less  and  less  thought  now  is  being  applied 
to  forecasting. 

Commercial  Weather  Services 

So  far  discussion  has  concentrated  on  public  and  military  weather  services.  What 
about  commercial  services?  The  growth  of  commercial  weather  services  indicates  that 
these  services  are  perceived  as  being  superior  to  the  service  routinely  available 
from  the  National  Weather  Service  public  forecasts  (Wallace,  1971  and  Gifford, 

1977).  As  an  example  of  the  rate  of  growth,  there  were  85  listings  in  the  certified 
consultant  and  professional  directories  in  the  December  1979  issue  of  the  Bulletin 
of  the  American  Meteorological  Society.  Wallace  (1971)  reported  there  were  28 
listings  in  the  December  1900  issue.  This  represents  an  increase  of  about  200 
percent  in  20  years  including  firms  that  do  not  provide  real  time  forecasting 
services.  Forecast  verification  data  for  these  enterprises  were  not  found  in  the 
literature;  however,  Suchman,  Auvine,  and  Hinton  (1979)  presented  an  evaluation 
of  the  benefits  of  meteorological  services  to  the  clients  of  one  firm.  Services  to 
road  and  street  departments,  electric  utilities,  and  gas  utilities  were  discussed. 

In  general,  a  single  accurate  forecast  of  a  significant  change  more  than  pays  for 
the  costs  of  the  services  for  one  year.  Similar  statements  have  been  made  by  Wallace 
(1971),  Freese  and  Notis  (1976),  and  in  the  Business  Week  Magazine  (1979).  Although 
it  is  clearly  established  that  the  major  motivation  for  subscribing  to  a  weather 
service  is  financial,  a  few  other  reasons  were  given.  The  importance  of  financial 
benefits  as  a  reason  for  buying  commercial  weather  service  logically  leads  to  strong 
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motivation  to  produce  accurate  forecasts.  Such  direct  strong  motivation  is  not 
present  in  forecasting  games  and  in  forecasting  facilities  not  directly  interfacing 
with  a  user. 

Forecaster  Motivation 

Forecaster  motivation  plays  a  major  role  in  enabling  the  commercial  weather  services 
to  successfully  give  the  forecast  accuracy  and  service  that  has  supported  growth  in 
both  number  and  size.  Forecaster  motivation  has  been  a  major  factor  in  the  good 
performance  of  the  Air  Heather  Service  forecasters,  and  it  was  a  major  factor  in  the 
improvement  of  the  Western  Region  forecasts  as  reported  by  Snellman  (1977)  and 
MacDonald  (1977).  Effective  motivation  programs  need  management  support  in  the  form 
of  helpful  technical  information,  feedback  of  verification  data,  and  consistent 
perception  and  recognition  of  exceptional  meteorological  work.  Air  Weather  Service 
distributes  publications  and  soundslide  seminars  tailored  to  forecasting  personnel; 
most  stations  verify  forecasts  daily,  and  a  centralized  verification  program  is 
being  prepared.  We  have  found  that  the  most  important  motivational  factor  is  unit 
leadership.  Occasionally  an  Air  Weather  Service  unit  will  experience  a  change  in 
leadership  without  any  change  in  the  forecasters.  In  some  of  these  cases,  an 
immediate  and  dramatic  change  in  forecasting  performance  has  occurred.  Interviews 
with  some  of  these  leaders,  and  others  who  maintained  good  forecasting  performance 
wherever  assigned,  revealed  that  the  main  common  factor  was  effective  motivation. 

As  an  example,  one  unit  had  the  lowest  performance  of  all  AWS  units  in  Europe. 

Within  a  few  months  after  a  new  commander  was  assigned,  this  unit's  performance  rose 
to  first  place  in  the  theater  and  remained  there  throughout  the  commander's  tour. 

The  key  was  the  interest  shown  by  the  commander  in  the  people  and  their  work.  We 
have  found  no  other  factor,  including  experience,  meteorological  education,  fore¬ 
casting  aids,  forecasting  guidance,  consultant  visits,  seminars  and  publications,  as 
effective  as  the  performance  typically  achieved  by  commanders  who  can  inspire 
(motivate)  the  forecasters  to  perform  well.  Clearly,  forecaster  motivation  is  an 
important  factor  in  forecasting  performance  and  skill. 

We  can  reasonably  assume  that  management  emphasis  was  not  a  major  factor  in  the 
university  games  nor  in  the  results  reported  by  Charba  and  Klein  (1980b).  In  the 
university  games  participation  generally  was  voluntary  and  extracurricular.  Eoth 
factors  are  consistent  with  the  aura  of  freedom  with  personal  responsibility  fostered 
by  universities.  This  does  not  mean  that  the  individuals  involved  are  inadequately 
motivated  or  treated  the  contest  lightly.  To  the  contrary,  the  author  understands 
from  a  former  student  participant  (W.  F.  Johnson,  Lt  Col,  USAF) ,  that  the  partic¬ 
ipation  is  serious  and  intense.  Nevertheless,  the  motivation  provided  by  a  poten¬ 
tially  bruised  ego  on  a  once  a  day  product  might  not  be  as  motivating  as  full  time 
forecasting  responsibilities  for  a  demanding  customer.  The  skill  achieved  was  quite 
high:  it  is  possible  that  the  skill  was  too  high  to  be  affected  by  the  improvements 
so  far  achieved  by  the  numerical  models.  Thus  it  is  possible  that  improvements  will 
be  achieved  later  with  further  improvements  in  large  scale  numerical  models. 
Breakthroughs  in  mesoscale  meteorology  and  methodology  would  also  help.  There  has 
been  little  published  concerning  management  emphasis  of  forecasting  quality  in  the 
NWS,  a  fact  which  coupled  with  a  few  articles  and  comments  to  the  contrary  (Stover, 
1974;  Golden,  et.  al.,  1978;  Roberts,  1978?  Schwartz,  1980)  lead  one  to  believe  that 
improvements  are  possible  through  encouragement  and  motivation  of  forecasters.  A 
note  of  caution;  motivational  programs  can  have  unwanted  side  effects.  We  under¬ 
stand  that  the  Western  Region  program  was  discontinued  because  of  them,  and  Air 
Weather  Service  has  had  to  adjust  its  program  several  times. 

Discussion 

The  traditional  approaches  for  improving  weather  forecasting  were  reviewed.  Improve¬ 
ments  in  data,  computers,  and  models  which  resulted  in  significant  increases  in 
forecasting  skill  for  forecasts  of  some  elements  were  not  achieved  across  the  board. 
Additional  factors  appear  to  be  important.  Univursity  meteorological  education  was 
not  found  to  be  essential  to  good  forecasting  performance.  In  fact,  beyond  funda-  'w 
mental  concepts,  no  correlation  was  found  between  amount  of  university  education 
and  forecasting  skill  except  for  one  study.  This  study  was  based  on  biographical 
characteristics  of  good  and  poor  officer  and  enlisted  forecasters  as  assessed  by 
peers.  The  good  forecasters  in  each  group  tended  to  have  more  university  education. 
However,  this  study  might  not  be  representative  of  forecasting  operations  in  the 
1980  base  weather  station. 


1980  base  weather  station.  Overall,  we  conclude  that  university  meteorological 
training  is  not  required  for  day  to  day  terminal  forecasting.  Chanute  training  is 
adequate,  provided  enough  university  trained  meteorologists  are  available  for  train¬ 
ing,  consultation,  and  guidance. 

Meteorological  experience  also  did  not  seem  to  be  an  important  factor  after  the  first 
month  or  two.  This  assessment  seems  to  apply  both  to  overall  forecasting  experience 
and  to  on  station  experience.  One  study  indicated  that  forecasters  who  began  fore¬ 
casting  prior  to  age  25  were  better.  Recently  many  forecasters  crosstrained  into  the 
weather  career  field;  often  the  crosstrainees  outperformed  the  others.  There  are,  of 
course,  some  poorer  performers  in  this  group,  but  they  seem  to  be  in  the  minority. 
There  were  a  few  indications  that  experienced  forecasters  tend  to  forecast  rare 
events  more  accurately.  However,  on  closer  inspection  the  instances  of  superior 
forecasting  could  be  attributed  to  instation  "education,"  meaning  OJT,  review  of 
local  case  studies  and  other  forecaster  reference  materials.  We  conclude,  that 
forecasting  experience  beyond  the  initial  one  to  two  months,  is  not  related  to  fore¬ 
casting  skill. 

Forecaster  aptitude  and  interest  appear  to  be  important.  These  attributes  generally 
seem  to  be  developed  during  various  stages  of  forecasters  careers.  Positive  managers 
seem  to  be  helpful  in  this  process.  Another  factor  considered  was  weather  station 
operations,  especially  the  degree  of  systematization  of  analysis  and  forecasting 
procedures  and  the  forecaster  workload.  In  practice,  development  of  systsmatia 
procedures  has  been  slow.  Progress  has  been  luade  for  those  forecaster  aids  produced 
completely  by  machine.  The  interpretation  of  these  aids  is  far  from  uniform;  so 
there  still  is  a  great  need  for  establishing  systematic  procedures.  For  example, 
some  of  the  MOS  forecasts  still  appear  to  have  greater  skill  than  manual  forecasts 
even  several  years  after  this  anomaly  was  first  published.  Forecaster  workload 
doesn't  appear  to  have  decreased  appreciably  in  spite  of  improvements  in  forecaster 
aids.  Forecasters  do  not  have  more  time  to  spend  on  thinking  through  forecasts, 
especially  in  poor  weather.  These  two  aspects  of  station  operations  both  are  desir¬ 
able,  but  data  showing  they  will  provide  an  incremental  increase  of  skill  are  lack¬ 
ing  . 

The  final  attribute  examined  was  forecaster  motivation.  The  rapid  increase  in  number 
and  size  of  private  sector  meteorological  service  companies  was  attributed  primarily 
to  the  monetary  incentive.  Forecaster  motivation  in  government  meteorological 
services  was  found  to  be  present  in  units  with  excellent  performance.  In  addition, 
in  case  after  case,  stations  that  had  long  records  of  poor  performance  achieved 
dramatic  improvement  after  assignment  of  a  new  officer  in  charge.  Interviews  with 
some  of  the  successful  leaders  indicated  simple  techniques  were  used  such  as  setting 
good  forecasting  as  a  unit  goal,  making  activities  directed  toward  this  goal  first 
[riority,  public  recognition  of  successful  individual  and  team  efforts,  and  Drovidino 
encouraging  guidance  when  needed.  It  was  suggested  that  forecaster  motivation  is 
the  major  one  that  determines  whether  the  state  of  the  practice  of  weather  forecasting 
is  comparable  to  the  state  of  the  science.  Motivation  was  most  effective  when 
a  mechanism  was  available  to  provide  prompt  feedback  of  forecasting  skill,  and  a 
program  established  to  make  pertinent  and  useful  technical  information  available. 
However,  motivation  can  be  counter  productive  if  used  without  sufficient  sensitivity 
to  the  needs  of  the  individual  forecasters.  No  other  factor,  attribute,  or  char¬ 
acteristic  has  been  found  which  is  more  important  to  high  forecasting  skill  than 
strong  motivation  of  the  forecasters  by  the  person  in  charge. 
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CONCLUDING  REMARKS 


This  investigation  was  begun  with  the  expectation  of  finding  that  our  overall 
weather  forecasting  performance  had  not  improved  recently,  primarily  because  of 
low  forecaster  experience  and  significantly  reduced  formal  meteorological  educa¬ 
tion  and  training.  Thus,  it  was  surprising  to  reach  the  conclusion,  on  the  basis 
of  available  data,  that  our  forecasting  performance  had  significantly  improved 
in  spite  of  decreased  forecaster  experience  and  education.  The  effects  of 
experience  and  education  apparently  are  realized  in  the  first  two  to  three  months 
of  active  forecasting.  This  finding  raises  several  questions  which  can  only  be 
answered  through  specifically  directed  research.  Topics  should  include  measuring 
the  forecasting  skill  of  highly  versus  minimally  experienced  and/or  educated 
forecasters,  validating  the  forecaster  selection  tests  identified  by  Kershner  (1967) , 
and  evaluating  the  effects  of  bonus  pay  for  high  performance,  less  forecaster  work¬ 
load,  and  inclusion  of  training  time  in  manpower  computations.  Inferences  made 
on  these  subjects  based  on  data  collected  30  years  ago  are  of  questionable  validity 
because  the  background  and  characteristics  of  the  typical  AHS  forecaster  are  so 
different  today.  Such  studies  would  be  useful  to  both  civil  and  military  organiza¬ 
tions,  and  could  be  conducted  by  either. 

The  conclusion  that  forecaster  motivation  is  the  single  most  important  factor 
determining  forecaster  skill  was  somewhat  disappointing  although  not  too  surprising. 
We  knew  it  was  important  because  of  our  professional  military  education  courses  and 
experience.  As  meteorologists,  we  would  have  liked  to  find  more  sensitivity  to  the 
science  of  meteorology  in  the  available  skill  measuring  techniques.  This  subject 
also  needs  further  research. 

In  general,  the  overall  result  of  this  effort  was  a  revalidation  of  the  cost 
effectiveness  of  Air  Weather  Service  emphasis  on  a  forecaster  force  mostly 
composed  of  enlisted  personnel.  It  indicates  that  corporately  we  may  be  overly 
concerned  with  forecasting  experience  and  formal  training,  and  insufficiently 
concerned  with  forecasting  motivation,  forecaster  selection,  and  enhancing  the 
thought  processes  required  for  forecasting.  The  AWS  emphasis  on  forecasting 
skill  apparently  was  effective,  and  should  be  continued.  Numerical  weather 
prediction  efforts  should  continue,  and  are  likely  to  improve  weather  forecasting 
over  the  next  few  years.  Further  improvements  likely  will  be  achieved  through 
continued  emphasis  on  weather  forecasting  skill  and  on  effective  programs  to 
encourage  forecasters  to  routinely  do  the  hard  mental  work  with  basic  meteorology 
that  is  essential  for  good  weather  forecasting. 
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conamai  over  climatology,  74 
to  96  hi'WU  (Boiart,  1975). 


tCMCSTCM 

Figure  6.  Percentage  improvement  over  SUM A  conteniui  temperature  lT) 
and  precipitation  (P)  forecast*  by  Or.  Boiart  (Bo«a*4,  1975). 
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figure  20.  NWS  ievere  weather  outlook 
ji aJLse  alarm  note.  (FAR),  probability 
of  detection  (POP),  and  critical  iucce** 
index  (CSI).  {Peanon  and  Wei**,  1019) 


Figure  21.  NWS  tornado  watch  percent 
correct  major  tornado  outhwakA  and 
three  year  Running  average  (crone* ) . 
(Pearion  and  lOeiii,  1919 ) 


FxguAt  tt.  Probability  of  precipitation,  NWS  We item  Peg  ion 
field  forecaster  (FP)  »t  M OS,  percent  improvement  over 
climatology,  December  through  Harch,  all  forecast  period*. 
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Figure  24.  Ceiling /visibility,  AblS  all  hours, 
peAce.nl  improvement  oveA  persistence  (German 
and  Hicks,  19 SOI. 
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Figure  26.  Same  as  FiauAe  t4  for  6  hours  only 
(German  and  Hicks, 
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Figure  25.  Same  as  FiguAe  24  jjo a  3  hours  only 
(German  and  Hicks,  198(3). 
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F-tguAe  27.  Same  as  Figure  24  ion  12  heart  onto 
(GeAman  and  Hicks,  19101. 
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